KEP-5593: Configure the max CrashLoopBackOff delay #5594

hankfreund · 2025-09-30T21:53:05Z

One-line PR description: Splitting KEP-4603: Tune CrashLoopBackoff into two KEPs.

Issue link: Configure the max CrashLoopBackOff delay #5593

Other comments: No material changes have been made to either KEP. Content was removed from one or the other and grammar has been updated to make sense.

k8s-ci-robot · 2025-09-30T21:53:14Z

Welcome @hankfreund!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-09-30T21:53:15Z

Hi @hankfreund. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tallclair · 2025-10-01T16:24:52Z

/ok-to-test

tallclair

Need to add a prod readiness file: keps/prod-readiness/sig-node/5593.yaml

k8s-ci-robot · 2025-10-01T20:05:44Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hankfreund
Once this PR has been reviewed and has the lgtm label, please assign derekwaynecarr, jpbetz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/prod-readiness/OWNERS
keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tallclair · 2025-10-02T05:56:32Z

keps/sig-node/5593-configure-the-max-crashloopbackoff-delay/README.md

+
+#### Beta
+
+- Gather feedback from developers and surveys


Do we have any feedback? I'm not sure we want to block the beta on this.

tallclair · 2025-10-02T05:59:05Z

keps/sig-node/5593-configure-the-max-crashloopbackoff-delay/README.md

+will rollout across nodes.
+-->
+
+<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>>


If we're targetting beta this release, this needs to be filled out. Or were you planning to cover this in a follow-up PR?

Risk is that a configured crashloop backoff causes the kubelet to become unstable. If that happens, a rollback just requires updating the config and restarting kubelet.

I wasn't sure initially if I should do it all in one, but it makes sense. Updated this and all the following sections.

tallclair · 2025-10-02T06:00:17Z

keps/prod-readiness/sig-node/5593.yaml

@@ -0,0 +1,3 @@
+kep-number: 5593
+alpha:
+  approver: TBD


This was already approved for alpha. You're just splitting the previous enhancement into 2 parts. I think you can put @soltysh here (copied from https://github.com/kubernetes/enhancements/blob/511963e97f955f97e9842ae3015b60af956539b3/keps/prod-readiness/sig-node/4603.yaml)

tallclair · 2025-10-02T06:01:57Z

keps/sig-node/5593-configure-the-max-crashloopbackoff-delay/README.md

+* `kubelet_pod_start_sli_duration_seconds`
+
+
+###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?


No, but I think you can just put N/A here. This feature is stateless.

tallclair · 2025-10-02T06:03:10Z

keps/sig-node/5593-configure-the-max-crashloopbackoff-delay/README.md

+
+<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>>
+
+###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?


tallclair · 2025-10-02T06:03:37Z

keps/sig-node/5593-configure-the-max-crashloopbackoff-delay/README.md

+previous answers based on experience in the field.
+-->
+
+###### How can an operator determine if the feature is in use by workloads?


tallclair · 2025-10-02T06:03:46Z

keps/sig-node/5593-configure-the-max-crashloopbackoff-delay/README.md

+
+<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>>
+
+###### How can someone using this feature know that it is working for their instance?


Observing the restart behavior?

lauralorenz

A few comments focused on clean separation between the two KEPs

lauralorenz · 2025-10-02T20:01:43Z

keps/sig-node/4603-tune-crashloopbackoff/README.md

 (Success) and the pod is transitioned into a "Completed" state or the expected
 length of the pod run is less than 10 minutes.

 This KEP proposes the following changes:


nit

Suggested change

This KEP proposes the following changes:

This KEP proposes the following change:

lauralorenz · 2025-10-02T23:26:53Z

keps/sig-node/4603-tune-crashloopbackoff/README.md

-
 Some observations and analysis were made to quantify these risks going into
 alpha. In the [Kubelet Overhead Analysis](#kubelet-overhead-analysis), the code
 paths all restarting pods go through result in 5 obvious `/pods` API server


I can't comment directly because its out of range of the diff but lines 499-508 contain references to the per Node feature still

I went through and I think I got all of them.

lauralorenz · 2025-10-02T23:38:50Z

keps/sig-node/4603-tune-crashloopbackoff/README.md

-included in the `config.validation_test` package.

 ### Rollout, Upgrade and Rollback Planning



On and after line 1149 (ref) in the Scalability section, the per Node feature is referenced, I suggest linking out to the other KEP there for inline context

lauralorenz · 2025-10-02T23:46:09Z

keps/sig-node/5593-configure-the-max-crashloopbackoff-delay/README.md

+This KEP proposes the following changes:
+* Provide a knob to cluster operators to configure maximum backoff down, to
+  minimum 1s, at the node level
+* Formally split image pull backoff and container restart backoff behavior


Should this bullet point still be in the other KEP as well since it was also done alongside that (or alternatively, should this bullet point be taken out of the top level content for both)? I think it was important to include references to these refactorings above the fold during the alpha phase so it was clear what was happening, but less important now that the alpha is implemented

I think removing it is all right.

lauralorenz · 2025-10-02T23:54:32Z

keps/sig-node/5593-configure-the-max-crashloopbackoff-delay/README.md

+rate limiting made up the gap to the stability of the system. Therefore, to
+simplify both the implementation and the API surface, this 1.32 proposal puts
+forth that the opt-in will be configured per node via kubelet configuration.
+


Now that this is the only feature referred to in this KEP, I feel like this section would read better with a subheading here like ### Implementing with KubeletConfiguration or something. Before it was all smooshed together since there were already so many H3s lol but that's not the case anymore

lauralorenz · 2025-10-03T00:01:16Z

keps/sig-node/5593-configure-the-max-crashloopbackoff-delay/README.md

+
+All behavior changes are local to the kubelet component and its start up
+configuration, so a mix of different (or unset) max backoff durations will not
+cause issues.


Just noticed that this sentence is kinda vague.

Suggested change

cause issues.

cause issues to running workloads.

lauralorenz · 2025-10-03T00:03:34Z

keps/sig-node/4603-tune-crashloopbackoff/README.md

-* Formally split backoff counter reset threshold for container restart backoff
-  behavior and maintain the current 10 minute recovery threshold
+* Provide an alpha-gated change to get feedback and periodic scalability tests
+  on changes to the global initial backoff to 1s and maximum backoff to 1 minute


Consider adding a sentence to both Overviews about how this was originally a bigger KEP that has been split out into two, and link to the other one there, so its quickly in context for new or returning readers.

lauralorenz · 2025-10-03T00:04:05Z

keps/sig-node/5593-configure-the-max-crashloopbackoff-delay/README.md

+* Formally split image pull backoff and container restart backoff behavior
+* Formally split backoff counter reset threshold for container restart backoff
+  behavior and maintain the current 10 minute recovery threshold
+


X-post from the other one: Consider adding a section to both Overviews about how this was originally a bigger KEP that has been split out into two, and link to the other one there, so its quickly in context for new or returning readers.

Tuning and benchmarking a new crashloopbackoff decay will take a lot of work. In the meantime, everyone can benefit from a per-node configurable max crashloopbackoff delay. Splitting the KEP into two KEPs to allow for graduating the latter to beta before the former.

This KEP is mostly a copy of keps/sig-node/4603-tune-crashloopbackoff with all the tuning bits removed (and grammar adjusted to make sense). The desire is to advance this KEP to beta sooner than we'd be able to advance the other one.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 30, 2025

k8s-ci-robot added the kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory label Sep 30, 2025

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 30, 2025

k8s-ci-robot requested review from derekwaynecarr and mrunalp September 30, 2025 21:53

k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 30, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 1, 2025

tallclair reviewed Oct 1, 2025

View reviewed changes

hankfreund force-pushed the clb_split_kep branch from cacae2d to 511963e Compare October 1, 2025 20:05

tallclair reviewed Oct 2, 2025

View reviewed changes

hankfreund force-pushed the clb_split_kep branch from 511963e to d7ce106 Compare October 2, 2025 14:12

lauralorenz reviewed Oct 3, 2025

View reviewed changes

hankfreund force-pushed the clb_split_kep branch 2 times, most recently from 1f49d2e to 1d0be4d Compare October 3, 2025 14:53

hankfreund added 2 commits October 6, 2025 22:03

hankfreund force-pushed the clb_split_kep branch from 1d0be4d to 684ee23 Compare October 6, 2025 22:11

		* `kubelet_pod_start_sli_duration_seconds`


		###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?


		<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>>

		###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?


		<<[UNRESOLVED beta]>> Fill out when targeting beta to a release. <<[/UNRESOLVED]>>

		###### How can someone using this feature know that it is working for their instance?

	This KEP proposes the following changes:
	This KEP proposes the following change:

		included in the `config.validation_test` package.

		### Rollout, Upgrade and Rollback Planning

KEP-5593: Configure the max CrashLoopBackOff delay #5594

Are you sure you want to change the base?

KEP-5593: Configure the max CrashLoopBackOff delay #5594

Uh oh!

Conversation

hankfreund commented Sep 30, 2025

Uh oh!

k8s-ci-robot commented Sep 30, 2025

Uh oh!

k8s-ci-robot commented Sep 30, 2025

Uh oh!

tallclair commented Oct 1, 2025

Uh oh!

tallclair left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Oct 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lauralorenz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!